The Multi-site 2009 EVALITA Spoken Dialog System Evaluation

This document presents the coordination and the evaluation procedures for the Spoken Dialogue System Task in EVALITA 2009. Three institutions participated into the competition, University of Trento, University of Naples and Loquendo. EVALITA participants were asked to develop a SDS application operating in the sales force domain, they were provided with a preliminary list of scenarios indicating system accounting modalities and a possible list of subtasks that should made possible. The three systems were hosted on a server at Trento, 19 volunteers called all of them. The calls have been recorded, transcribed and annotated. The evaluation work, based on scripting run on the annotations, has been mainly focused on assessing performance at the dialogue, task, and concept levels. Detailed results indicating the systems performances are reported in the paper. This document presents the coordination and the evaluation procedures for the Spoken Dialogue System Task in EVALITA 2009. Three institutions participated into the competition, University of Trento, University of Naples and Loquendo SpA.